Deployment Revert Process

Overview

This document describes how to revert to a previous deployment in production for the Webapp, Backend and Workers services.

caution

Reverting a production deployment is rare and should be carefully considered. Always discuss with the team before proceeding. We have

WebApp (nextjs+vercel)

Deployment is done automatically by Vercel on push on production branch. To revert the deployment to previous deployed version:

1. Vercel production webapp deployments
2. Find the previous deployment we want to revert to > click on it
3. On the single deployment page > 3 dots > Redeploy

Notice: Vercel support Instant Rollback option, which restores the selected deployment. We choose not to use this option for 2 reasons:

1. All environment variables, cron jobs, etc., revert to that deployment’s state
2. Vercel will turn off auto-assignment of production domains. This means that when you or your team push changes to production, the roll backed deployment won't be replaced.

Backend and Workers

Deployment is done using github action triggered when merging to production branch. The action checkout the code, build the docker image, push it to ECR and deploy the image (for worker no deployment as we run the docker container on demand)

To revert the deployment to previous deployed version:

Backend - Github UI

1. Backend repo deploy action
2. Find the relevant previous deployment (you can identify by commit hash or timestamp) > click on it
3. Re-run all jobs - it will re-run the action on the same commit which will result is deployment of this version

Worker - Github UI

1. Investigation worker repo deploy actions
2. Find the relevant previous deployment (you can identify by commit hash or timestamp) > click on it
3. Re-run all jobs - it will re-run the action on the same commit which will result is deployment of this version

---

🚨 Bonus: In-Production Revert Guide (Not Recommended)

Warning: This emergency-only process is extremely rare and should be carefully considered. Always discuss with the team before proceeding.

Use this only if all other recovery options have been exhausted.

---

1. Revert in Production

You have two rollback options in production:

1.a. Revert a Single Commit

GitHub UI

1. Select the production branch in your repo.
2. Find the bad commit in Commits, click Revert.
3. GitHub creates a temporary revert branch.
4. Open a PR from that branch → production, review, then merge.

Command Line

# New branch for revert
git checkout -b revert-bad-commit origin/production
# Revert commit
git revert <SHA>
# Push branch
git push origin revert-bad-commit

• Then open a PR revert-bad-commit → production and merge.

1.b. Revert an Entire Deployment PR

1. On the merged deployment PR (main → production), click Revert.
2. GitHub generates a new PR that undoes the original.
3. Review and merge this PR into production.

---

2. Merge the Revert into main

1. Create a branch:

   git checkout -b merge-revert-into-main origin/main
2. Merge updated production:

   git fetch origin
   git merge origin/production 
3. Resolve conflicts, commit, then:

   git push origin merge-revert-into-main:<remote branch> 
4. Open PR → main, review, merge.

---

3. Revert the Revert on main

Restore the feature:

git checkout -b restore-feature
git revert <REVERT_SHA>
git push origin restore-feature

• Open PR restore-feature → main, review, merge.

---

4. Fix the Original Bug (Re-apply with a Fix)

git checkout -b fix-production-issue
git add .
git commit -m "Fix: [describe the fix]"
git push origin fix-production-issue

• Open PR fix-production-issue → main, review, merge.

---

5. Create PR Back to production

1. Open PR main → production.
2. Describe revert, restore, fixes.
3. Assign reviewers & merge.